<!DOCTYPE html>
<html>

<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
	<meta name="theme-color" content="#33474d">
	<title>Memcached数据被踢(evictions0)现象分析 | 失落的乐章</title>
	<link rel="stylesheet" href="/css/style.css" />
	
      <link rel="alternate" href="/atom.xml" title="失落的乐章" type="application/atom+xml">
    
</head>

<body>

	<header class="header">
		<nav class="header__nav">
			
				<a href="/archives" class="header__link">Archive</a>
			
				<a href="/tags" class="header__link">Tags</a>
			
				<a href="/atom.xml" class="header__link">RSS</a>
			
		</nav>
		<h1 class="header__title"><a href="/">失落的乐章</a></h1>
		<h2 class="header__subtitle">技术面前，永远都是学生。</h2>
	</header>

	<main>
		<article>
	
		<h1>Memcached数据被踢(evictions0)现象分析</h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/Memcached/">Memcached</a>
			</span>
		
	</div>

	

	
		<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;很多同学可能熟知Memcached的LRU淘汰算法，它是在slab内部进行的，如果所有空间都被slabs分配，即使另外一个slab里面有空位，仍然存在踢数据可能。你可以把slab理解为教室，如果你的教室满了，即使别的教室有空位你的教室也只能踢人才能进人。</p>
<p><figure class="figure"><img src="https://github.com/hcldirgit/image/blob/master/Memcached%E6%95%B0%E6%8D%AE%E8%A2%AB%E8%B8%A2evictions0%E7%8E%B0%E8%B1%A1%E5%88%86%E6%9E%90/01.png?raw=true" alt=""></figure></p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;本文介绍的却是另外一种现象。今天监控发现线上一memcached发生数据被踢现象，用stats命令看evictions&gt;0,因为以前也出现过此问题，后来对这个参数增加了一个监控，所以这次主动就发现了。由于给memcached分配的内存远大于业务存储数据所需内存，因此初步判断是“灵异现象”。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;第一步，netstat查看所有连接，排除是否被一些未规划的client使用，经排查后断定无此可能。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;第二步，用tcpdump抽样检查set的指令，排除是否有忘记设cache过期时间的client，初步检查所有典型的业务都有expire time。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;第三步，Google，未果</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;第四步，看源代码，了解evictions计数器增加时的具体细节.</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;in items.c, memcached-1.2.8,</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">for</span> (search = tails[id]; tries &gt; 0 &amp;&amp; search != NULL; tries--, search=search-&gt;prev) &#123;</div><div class="line">    <span class="keyword">if</span> (search-&gt;refcount == 0) &#123;</div><div class="line">        <span class="keyword">if</span> (search-&gt;exptime == 0 || search-&gt;exptime &gt; current_time) &#123;</div><div class="line">            itemstats[id].evicted++;</div><div class="line">            itemstats[id].evicted_time = current_time - search-&gt;time;</div><div class="line">            STATS_LOCK();</div><div class="line">            stats.evictions++;</div><div class="line">            STATS_UNLOCK();</div><div class="line">        &#125;</div><div class="line">        do_item_unlink(search);</div><div class="line">        <span class="built_in">break</span>;</div><div class="line">    &#125;</div><div class="line">&#125;</div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;从源代码发现踢数据只判断一个条件，if (search-&gt;refcount == 0)，这个refcount是多线程版本计数用，在当前服务器未启用多线程情况下，refcount应该始终为0,因此初步判断memcached是从访问队列尾部直接踢数据。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;为了证实想法，设计以下场景：</p>
<ol>
<li><p>部署一个memcached测试环境，分配比较小的内存，比如8M</p>
</li>
<li><p>设置1条永远不过期的数据到memcached中，然后再get一次，这条数据后续应该存在LRU队尾。</p>
</li>
<li><p>每隔1S向memcached set(并get一次) 1,000条数据，过期时间设为3秒。</p>
</li>
<li><p>一段时间后，stats命令显示evictions=1</p>
</li>
</ol>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;按我以前的理解，第2步的数据是永远不会被踢的，因为有足够过期的数据空间可以给新来的数据用，LRU淘汰算法应该跳过没过期的数据，但结果证实这种判断是错误的。以上业务的服务器发生被踢的现象是由于保存了大量存活期短的key/value,且key是不重复的。另外又有一业务保存了小量不过期的数据，因此导致不过期的数据惨遭被挤到队列踢出。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;本来这个问题就告一段落了，但在写完这篇文章后，顺便又看了新一代memcached 1.4.1的源代码，很惊喜发现以下代码被增加。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;items.c, memcached 1.4.1<br>/<em> do a quick check if we have any expired items in the tail.. </em>/<br>int tries = 50;<br>item *search;</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">for</span> (search = tails[id];</div><div class="line">    tries &gt; 0 &amp;&amp; search != NULL;</div><div class="line">    tries--, search=search-&gt;prev) &#123;</div><div class="line">    <span class="keyword">if</span> (search-&gt;refcount == 0 &amp;&amp;</div><div class="line">        (search-&gt;exptime != 0 &amp;&amp; search-&gt;exptime &lt; current_time)) &#123;</div><div class="line">        it = search;</div><div class="line">        /* I don<span class="string">'t want to actually free the object, just steal</span></div><div class="line"><span class="string">         * the item to avoid to grab the slab mutex twice ;-)</span></div><div class="line"><span class="string">         */</span></div><div class="line"><span class="string">        it-&gt;refcount = 1;</span></div><div class="line"><span class="string">        do_item_unlink(it);</span></div><div class="line"><span class="string">        /* Initialize the item block: */</span></div><div class="line"><span class="string">        it-&gt;slabs_clsid = 0;</span></div><div class="line"><span class="string">        it-&gt;refcount = 0;</span></div><div class="line"><span class="string">        break;</span></div><div class="line"><span class="string">    &#125;</span></div><div class="line"><span class="string">&#125;</span></div></pre></td></tr></table></figure>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;重复进行上述测试，未发生evictions。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;9/8 Update: 注意到L108的tries=50没有？试想把测试第2步设置51条不过期数据到cache中，情况会怎样？因此新版的Memcached也同样存在本文描述问题。</p>
<p>&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;几条总结：</p>
<ul>
<li>过期的数据如果没被显式调用get，则也要占用空间。</li>
<li>过期的不要和不过期的数据存在一起，否则不过期的可能被踢。</li>
<li>从节约内存的角度考虑，即使数据会过期，也不要轻易使用随机字符串作为key，尽量使用定值如uid，这样占用空间的大小相对固定。</li>
<li>估算空间大小时候请用slab size计算，不要按value长度去计算。</li>
<li>不要把cache当作更快的key value store来用, cache不是storage。</li>
</ul>

	

	
		<span class="different-posts"><a href="/2017/10/12/Memcached/4. Memcached数据被踢(evictions0)现象分析/" onclick="window.history.go(-1); return false;">⬅️ Go back </a></span>

	

</article>

	</main>

	<footer class="footer">
	<div class="footer-content">
		
	      <div class="footer__element">
	<p>Hi there, <br />welcome to my Blog glad you found it. Have a look around, will you?</p>
</div>

	    
	      <div class="footer__element">
	<h5>Check out</h5>
	<ul class="footer-links">
		<li class="footer-links__link"><a href="/archives">Archive</a></li>
		
		  <li class="footer-links__link"><a href="/atom.xml">RSS</a></li>
	    
		<li class="footer-links__link"><a href="/about">about page</a></li>
		<li class="footer-links__link"><a href="/tags">Tags</a></li>
		<li class="footer-links__link"><a href="/categories">Categories</a></li>
	</ul>
</div>

	    

		<div class="footer-credit">
			<span>© 2017 失落的乐章 | Powered by <a href="https://hexo.io/">Hexo</a> | Theme <a href="https://github.com/HoverBaum/meilidu-hexo">MeiliDu</a></span>
		</div>

	</div>


</footer>



</body>

</html>
